IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In Re Patent Application of: ORELL et al . 
Serial No. : 10/035, 321 

Filed: January 4, 2002 

For: STREAMING AND MANAGING COMPLEX MEDIA CONTENT ON 

WEB SERVERS 

Group Art Unit: 2144 
Examiner: Greg C. Bengzon 

RULE 132 DECLARATION OF ZOHAR SIVAN 

I, the undersigned, Zohar Sivan of 13 Odem Street, 
Zichron Yaakov, Israel, hereby declare as follows: 

1. Dror Orell and I are the Applicants in U.S. 
Patent Application No. 10/035,321 (hereinafter "the 
Application") . 

* 

2. In an Office Action dated April 15, 2005, claims 1- 
32 in the aforesaid Application were rejected under 35 
U.S.C. 102(e) over U.S. Patent 6,848,004, to Chang et al. 
(hereinafter "the Chang Patent"). In making this 
rejection, the Examiner cited material in cols. 8-9 of 
the Chang Patent regarding the use of a servlet program 
in extracting and delivering content of HotMedia files to 
a client station. The Chang Patent does not claim the 
use of a servlet for this purpose or any other purpose. 

3. Prior to July, 1999, Dror Orell and I conceived and 
began development of the invention claimed in the 
Application. We are the inventors of the subject matter 
relating to the use of servlets that was disclosed in the 
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Chang Patent. 

4. Our invention was presented by Hagai Krupnik, a 
member of our group at IBM Haifa Research Laboratory, at 
a meeting held with a group at the IBM Watson Research 
Center, on or about July 1, 1999 (hereinafter "the Watson 
Group ") . The Watson Group included a number of the 
inventors of the Chang Patent, including Chang, Kumar, 
Lipscomb and Zhang. Following the meeting we submitted a 
proposal for development of "HotMedia Streaming Servlet 
APIs" to the Watson Group. It appears that the inventors 
of the Chang Patent learned of the possible use of 
servlets in their HotMedia architecture from this meeting 
and proposal . 

5. In support of my Declaration, Appendix A hereto 
contains, the proposal we f submitted to the Watson Group. 
Appendix B contains an e-mail message, retrieved from the 
IBM Lotus Notes database, that was sent by Hagai Krupnik 
on July 15, 1999, to members of the Watson Group 
regarding our proposal. Appendix C contains another e- 
mail message that Subrina Chang sent to Hagai Krupnik 
acknowledging the proposal. 
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In Re: U.S. S.N. 10/035,321 
Group Art Unit 2144 

Rule 132 Declaration of Zohar Sivan cont'd 

I hereby declare that all statements made herein of my 
own knowledge are true and that all statements made on 
information and conjecture are thought to be true; and 
further that these statements were made with the 
knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment, or both, 
under Section 1001 of Title 18 of the United States Code 
and that such willful false statements may jeopardize the 
validity of the application of any patent issued thereon. 




Zohar Sivaw, Citizen of Israel 
13 Odem Street, Zichron Yaakov, Israel 



Date: 
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APPENDIX A 



HotMedia Streaming Servlet APIs 

proposal 

This document contains a draft proposal for APIs for a servlet that will support 
HotMedia content streaming. 

The HotMedia Streaming Servlet will be a standalone module running behind/on top 
of a standard http server. Hence the only interaction with other HotMedia modules is 
via URL requests that will be sent from HotMedia applet to the http Server. 
Therefore, the servlet's functionality is to be defined through the format/syntax of 
URLs which it should respond to, and the content of the http response. 

The task of this document is to define this URL syntax. This document does not 
discuss the changes which are required at the applet side (HotMedia Player) in order 
to make use of the servlets module. While this does not seem to have impact on the 
server side it is advised to design the required changes at early stages, (see remark 
below) 

The servlet definition in this document applies only to the first phase of intducing 
servlets to the HotMedia architecture. This phase is restricted to a "Stateless" 
implementation. The servlet will not use a "session manager", hence no "Back 
channel" is provided to support real time/online modification of a previous MVR 
stream request. However some online capabilities can be achieved by means of 
closing the current MVR file and reoppening it with new streaming parameters. The 
question how to use it within the HotMedia Applet should be addressed elsewhere. 

If and when a session manager will be added to the servlet module, it will be 
"backwards compliant". I.e. Previous HotMedia Applet will be able to use the "new" 
hotMedia Streaming Servlet. 

Document version : 0.03 July 15 Servlet Name : We shell refer to this 
module as HotMedia Streaming Servlet 

Servlet Module Requirements: 

The servlet module will be a set of pure Java classes. The servlet's "main" class name 
will be hmServlet.class. The "instalation" of this servlet should be simply to put a set 
of classes in a predefine location within the servlet-engine directory tree. 

The servlet module will have to be tested such that it is working properly across main 
http servers on several platforms. The test plan will be defined later. 

Since the servlet has to parse mvr file format it is desirable if the servlet can have a 

parser which is aligned with the HotMedia Player parser (see remark 

below) 



Servlet URL calls : 



To retrieve mvr stream via hmServlet, the URL that will be sent from hmMaster will 
be of the following Syntax : 

http://hostoame/sen4e<yhmSen4et / mwFilePath?queryString 

where : 

■ The red part is fixed and hard coded 

■ hostname is the name of the web server 



nivrFilePath is the path to the mvr file including its filename + 
extension and excluding the root 

E.g. if an mvr file can be downloaded from an http server 
(without servlet support) using the URL : 

http://MyWebServer/M^ 
than the servlet URL request will be : 

http://MyWebSercer/servte^ 
File.mvr?queryString 

■ quervString is a string which uses standard SQL format as 
described below. 

■ Due to servers inconsistencies - all queries are assumed to be 
case sensitive. 

■ calling http://hostname/servlet/hmServlet/ without MVR 
filePath will be used for remote debugging. 



Using such a call directly from the URL window of a browser 
will create a text/html file response with various variables that 
will be helpful for debugging (hotMediaServlet version, some 
data on the server machine). 

■ Calling http://hostname/servlet/hmServleiymvrFilePath without 
a query string will be used for debugging as well. 

In this case the mvr file will be sent to the client "as is", hence 
the servlet will serve as an "Identity filter". 
This feature can be used later on as a preformance checker to 
verify that the increase of delay & servers CPU consumption 
due to servlet module is minimal. 
The http response for these request will be of mime type "application/octet-stream" . 
This type is "basic" and works well when an applet tries to read binaries input stream 
from an http server. To our best knowledge, this detail has no effect on any other 
component in the framework. (Le. nothing has to be configured to support this). 



Servlet queryString : 

The servlet queryString is of the following format : 

servlet Actioii=actiori/&sei^ 
where : 

■ All sub strings action/, paramz & value; should be URL 
encoded prior to concatenation (see sample code) 

■ servletActionfs) are the name of the actions that the servlet is 
requested to perform. Typically only one action will be used 
per call. However this is not mandatory (see details below). 

=> If no action is selected the query string will be null and the 
mvr file will be sent to the client "as-is". 
=> some actions can not be performed simultaneously while 
some do not (see details below) 

■ param/ is the name of the i f th* parameter that is required in 
order to perform the action (see details below) 

* The parameters can be ordered arbitrarily. 

■ value/ is the value of the i'th* parameter that is required in 
order to perform the action (see details below) 

* The parameters can be ordered arbitrarily. 

There is a delimiter between each parameter/value pair 

■ No spaces will be present in any of the 
actions/parameters/values unless they are strings which are 
taken from the mvr headres. In this case - theservlet will 
support whatever hotMedia applet support. 

Suggested Servlet list of action : 



1. 


startAtBvte 


2. 


selectTime 


3. 


selectTracks 


4. 


skipTracks 


5. 


selectMediaTvpe 


6. 


skipMediaTvpe 


7. 


FitToBandwidth 


8. 


1 imitStreamSpeed 


9. 


getZoomedlmaee 



10. andup-TBD 



Note : Not all of the above will be implemented in the coming HotMediaTurbo Beta 
release. 



For each action we now specify the parameters that can be used and a list of actions 
that can not be handled in a single URL request (prohibited simultaneous Actions). 

1. startAtByte - this action is a "dumb" servlet action which simply start 

streaming from an arbitrarily requested byte within the MVR file. It is the duty 
of the hmApplet (hmMaster?) to calculate the start point for the servlet stream 
and to handle the coming stream accordingly. The output is not a valid mvr 
stream. 



parameters : startByte=mt Value where int Value is a string that represent 
a non-negative integer which is the location of the first byte within the MVR 
file. 

prohibited simultaneous Actions : selectTracks, skipTracks selectMedia. 
skipMedia, FitToBandwidth, 

output : A bitstream that makes sense only to the sender since it starts from 
an arbitrary point within the 



2. selecfTime - this action will start streaming from an arbitrarily requested time 
within a specific track within the MVR file. The output is a valid mvr stream 
which holds only portion of the track that is selected via the parameters. 



Note that it might be suitable to call the hmPlayer startOfData() following the 
URL request of this action. 

parameters : trackld=track.I.dValue where trackldValue is a string that 
represent the trackld to be selected 

startTime=int Value where intValue is a string that represent 
a non-negative integer which represents the the requested time in msec from 
which the servlet should stream this track. 

Note : if startTime parameter is missing from the Query 
string the servlet will send the bitstream from the beginning of the selected 
track 

endTime=int Value where intValue is a string that represent 
a non-negative integer which represents the the requested time in msec from 
which the servlet should stop streaming this track. 

Note : if endTime parameter is missing from the Query string 
the servlet will send the bitstream till the end of this track 



prohibited simultaneous Actions : selectTracks, skipTracks selectMedia, 
skipMedia, 



3. output : The output is a valid mvr stream which adjust all its headers to 
correctly define the selected portion of the selected track. 



4. selectTracks - This action specifically selects the tracks that should be 
streamed to the client, all other tracks are skipped. 



parameters : crackId=trackIdValue where trackldValue is a string that 
represent the trackld to be selected 

Note : there could be several <trackId=trackIdValue > pairs in 
a single servlet call. 

prohibited simultaneous Actions : skipTracks, selectMediaType, 
skipMediaType, .startAtByte 

output : A valid mvr stream which is a subset of the original mvr file. 



5. skipTracks - This action selects the tracks that should be skipped when the 
mvr file is streamed to the client. All other tracks are sent by their order. 



parameters : trackld=trackld Value where trackldValue is a string that 
represent the trackld to be selected 

Note : there could be several <trackld=trackld Value > pairs in 
a single servlet call. 

prohibited simultaneous Actions : selectTracks, selectMediaType, 
skipMedia Type, start AtByte 

output : A valid mvr stream which is a subset of the original mvr file. 



6. selectMediaType - This action selects the tracks that corresponds to a 

specific media type and stream them to the client. All other tracks are skipped. 

parameters : MediaType=MediaTypeString where MediaTypeString is a 
string that represent a hotMedia valid MediaType 

Note : there could be several <MediaType=MediaTypeString> 
pairs in a single servlet call 

prohibited simultaneous Actions : selectTracks, skipTracks, 
skipMediaType, startAtByte 

output : A valid mvr stream which is a subset of the original mvr file. 



7. skipMediaType - This action selects the tracks that should be skipped when 
the mvr file is streamed to the client. All other tracks are sent by their order. 



parameters : MediaT\"pe=MediaTypeString where MediaTypeString is a 
string that represent a HotMedia valid MediaType 

Note : there could be several <MediaType=MediaTypeString> 
pairs in a single servlet call. 

prohibited simultaneous Actions : selectTracks. skipTracks, 
selectMediaType, startAtByte 

output : A valid mvr stream which is a subset of the original mvr file. 



8. FitToBandwidfh - This is a recommendation to the servlet to fit the MVR 
representation to a specific bandwidth. The discussion of how will the server 
use it is postponed to the future (and so are the parameters). 



parameters : TBD 

prohibited simultaneous Actions : startAtByte (may be others as well 
TBD) 

output : A valid mvr stream which is a subset of the original mvr file. 



9. limf tStreamSpeed - This action recommends to the server to send the stream 
at a speed no faster than a specified speed. 

Typically an http server streams the data As fast as possible. However when 
the client sends several requests symultanousely (such as request for classes, 
request for other html pages etc..) and the client's bandwidth is limited it 
might be desirable to limit the speed of the mvr streaming. This action is 
provided in order to leave enough bandwidth to other streams that the client 
opens such as streams to get classes and plain http streams. The output stream 
content is not changed by this action 

Note : At this stage we shell consider this action as exploratory since more 
has to be done to verify that the client can really benefit from such approach. 

parameters : streamSpeed=int Value Speed where intValueSpeed is a string 
representation of an integer value which is the preferred speed in kBit/sec 

prohibited simultaneous Actions : none 

output : The output stream content is not changed by this action 



10. getZoomedimage - This action provides an interface to high resolution 
Zoom View of a given Image. 

In this implementation it is assumed that : 

=> There are two initial images with the same content in different resolution 
denoted inline image and High resolution image 

=> The two images are of the same aspect-ratio (width/height) up to some 
rounding error. 

=> The client/applet has the capabilities to display the inline image as a 
whole, and then mark a portion of the image of which the user would like to 
magnify.The look and feel (GUI) of the portion selection is irrelevant to the 
request as long as the applet is able to describe the selected portion using the 
parameters described below. 

The servlet calculates which portion of the high Resolution it should crop, and 
how much to down scale it such that the result will have identical range to the 
inline image crop and identical size to the whole inline image 

parameters : 

inlineWidth=intValueString - The width (in pixels) of the 
Inline Image to be magnified 

inlineHeight=intValueString - The height (in pixels) of the 
Inline Image to be magnified 

width=in.tValueString - The width (in pixels) of the selected 
portion of the Inline Image to be magnified 



hei ght=int V alueString - The height (in pixels) of the selected 
portion of the Inline Image to be magnified 

xHniValueString - The x coordinate (in pixels) of the upper 
left corner of the selected portion of the Inline Image to be magnified 

y=intValueString - The y coordinate (in pixels) of the upper 
left corner of the selected portion of the Inline Image to be magnified 

imageFilePath=FileSlring - A path to either a high resolution 
JPEG image or an MVR file which includes such high-resolution Image. 

trackId=trackJdSti*ing - Used only if If the High resolution Image 
is within an MVR file. The trackld of the High resolution image 

Note : If we will need to fetch the high-resolution images from 
a Database the imageFilePath parameter will have to be generalize to support 
the needed info for fetching images from the DataBase. However the issue of 
handling images that are stored in databases is beyond the scope of this 
document. 

prohibited simultaneous Actions : selectTracks. skipTracks, 
selectMediaType, skipMediaType, startAtByte, 

output : The output stream content is either an MVR stream which has a 
single media track holding a JPEG image, or a "stripped" JPEG stream (issue 
TBD) 

The dimension of the output image is inline WidthXinlineHeight 

pixels 



Error handling : 

When a servlet parses a request it might find out that it can not execute the request. 
Reasons for this may vary: 

• MVR File can not be read (absent/permissions etc..) 

• queryString is not valid/can not be handled. 

• MVR file has bugs/unsupported features etc.... 

• runtime errors (outOfMemory, bad connection etc ...) 

In terms of error handling, the errors in the servlet can be separated into two types of 
errors : 

1. Errors that the servlet identifies before it starts streaming data to the client. 

2. Errors that occur during the servlet's streaming. 

1 . In the first case the server can signal the problem to the applet using the http 
header fields. 

For example the httpResponseCode could be changed from the typical 200 to 
some redefined code number. This can be detected by the applet using 
URLConnection.getHeaderField(....). Moreover - in this case the servlet can 
stream an html/text file which describe the problem .... (useful for debugging) 



2. In the second case - the only thing that the servlet can do is to close the output 
stream. This, in turn will cause an exception in the applet. 

Additional remarks : 

1 . We have to analyze the changes needed to be done in the client side at an early 
stage and make sure that the described URL calls : 

o calls can serve HotMedia applet properly. 

For example : For each request we have to ask were can we fire such a 
request from the current/modified HotMedia Applet 

o can be created easily within HotMedia Player ? 

For example : For each request we have to ask ourselves how will 
hotmedia be capable to create the needed parameters for the required 
action(s). 

2. Can we have an mvr parser class that will be common to the applet and the 
servlet ? Currently the parser is part of hmMasket and it is not trivial to 
separate its source from the rest of the Java file. 



A code example : 

Here is a sample (pseudo) code that should be integrated into the applet in order to 
create a URL request to the HotMedia Streaming Servlet 
Note : This pseudo code has not been complied and (naturally) not tried as a 
standalone. Therefore it is not guarantied to be bug free ... 



String ServletAction [ ] = 

String param [ ] = 

String value [ ] = 

URL mvr URL = new 

URL { "http : / /myHost /myContentPath/myMvrFile . mvr" ) ; 

// 

// absolute & complete URL definition 
String servletPath - 

mvrURL.getProtocol () +mvrURL . get Host ( ) +"servlet/hmServlet/"+mvrU 

RL. get File () ; 

String queryString=" ?" ; 

queryString+= URLEncoder . encode (" servletAction" ) +"=" 
+URLEncoder . encode (ServletAction [i] ); 
for( ) { 

queryString+= "&"+ URLEncoder . encode { "servletAction" ) +" = " 
+URLEncoder . encode ( ServletAction [ i ] ) ; 
} 

for< ) { 



queryString+=" &" + URLEncoder . encode (param [ i ] ) +"=" 
-f-URLEncoder . encode (value [i ] ) ; 
} 

URL url = new URL ( servletPath+queryString) ; 
uric = url . openConnection ( ) ; 

uric . setusecaches ( false) ; //All servlet calls should not be cached in 
the browser cache. 

dis = new DatalnputStream (uric . get InputStream ()) ; 
// 

// from here on one can use the dis as if it is a regular dis 
from an http response. 



Comments, suggestions and discussions are more than welcomed : 
Hagai Krupnik 

Audio/Video Group - Multimedia Technologies 

IBM Research Laboratory in Haifa, Israel. 

LotusNotes : Hagai Krupnik/Haifa/IBM@IBMIL@IBMHAIFA 

Phone : +972-4-8296436 

Fax : +972-4-8296112 

snail Mail: MATAM Haifa 31905 ISRAEL. 

Email : hasai(a),iL ibm.com 
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Hagai 
Krupnik/Haifa/IBM 

To 

19:20 15/07/1999 Jeane Chen/Watson/IBM@IBMUS 

Keeranoor Kumar/Watson/IBM@IBMUS , 
Robert Plotkin/Watson/IBM@lBMUS , 
Subrina Chang/Watson/IBM@IBMUS , 
Liang-Jie Zhang/Watson/IBM@IBMUS , 
James Lipscomb/Watson/IBM@IBMUS , 
Peter Westerink/Watson/IBM@IBMUS , 
Ed SnibIe/Watson/IBM@IBMUS, Gabriel 
Taubin/Watson/IBM@IBMUS, Yun 
Deng/China/IBM@IBMCN, William 
Gaddy/Watson/Contr/IBM@IBMUS 
cc 

Gilad Cohen/Haifa/IBM@IBMIL, Ehud 
Karnin/Haifa/IBM@IBMIL@IBMDE / Zohar 
Sivan/Haifa/IBM@IBMIL 

Subject 

HotMedia Streaming Servlet APIs 
proposal 



Hi all 

It's been two weeks since our summit meeting. Personally IVe enjoyed the 
meeting and it was nice to meet in person people I've only recognize by 
their notes header. 

Anyway, now we're back to business and there is a "Architecture complete 
deadline" in two weeks from now. So.... 

I have just posted our proposal for HotMedia Streaming Servlet APIs to the 
HotMedia discussion database. It is very important that we'll get a serious 
feedback on it. Even though, I can guess, most of you are busy with other 
aspects of HotMedia/HotMediaTurbo. I just want to make sure that the 
designed servlet is really useable for HotMedia applet, otherwise we should 
alter the definition before we get into coding. If there is anyone who 
can't access this database, please let me know and I'll mail him/her the 
file directly. 



looking forwards to hear your comments/suggestions 
Hagai Krupnik 

Audio/Video Group - Multimedia Technologies 
IBM Research Laboratory in Haifa, Israel. 



LotusNotes : Hagai Krupnik/Haifa/IBM@IBMIL@IBMHAIFA 

Email : hagai @ il.ibm.com 

Phone : +972-4-8296436 

Fax : +972-4-8296112 

snail Mail : MATAM Haifa 31905 ISRAEL. 
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-Forwarded by Hagai Krupnik/Haifa/IBM on 07/20/99 
09:41AM 



To: Hagai Krupnik/Haifa/IBM@IBMIL@IBMDE 
cc: Jeane Chen/Watson/IBM@IBMUS 
From: Subrina Chang/Watson/IBM@IBMUS 

Subject: Re: HotMedia Streaming Servlet APIs proposal (Document link: 
Hagai Krupnik( 

Hagai, 

Thanks for your effort to come up this document. We will need some time to 
review your proposal. We will get back to you as soon as possible. 

Subrina 



Subrina Chang 

IBM - Internet Media Group 

Tel: 914-784-5756 TieLine: 863-5756 

e-mail: subrina@us.ibm.com 



